#项目1 数据采集技术概述  
使用 apache httpd、filebeat、logstash、csv完成数据采集演示。  
####1、apache httpd  
___
####1.1 目录信息  
运行目录：/app/httpd/  
日志目录：/app/httpd/logs/access_log  
配置文件：/etc/httpd24/httpd.conf  

####1.2 步骤  
1） 验证 httpd 运行 在浏览器里访问：http://127.0.0.1  
####2、filebeat  
___
####2.1 目录信息  
安装路径：/bigData/tools/filebeat/  
配置文件：/home/config/filebeat.yml  

####2.2 步骤  
1）修改配置文件  
```
filebeat.inputs: 
- type: log 
  enabled: true
  paths:
    - /app/httpd/logs/access_log

output.logstash:
  hosts: ["localhost:5044"]
```
2）启动：  
2.1）进入filebeat目录  
```
cd /bigData/tools/filebeat/
```
2.2）启动filebeat  
```
filebeat -c /home/config/filebeat.yml
```
####3、logstash  
___
####3.1 目录信息  
安装路径：/bigData/tools/logstash/  
配置文件：/home/config/logstash-filebeat-csv.conf  
```
input { 
    beats {
         port => 5044
    }
}
filter {          
     grok {
          match => { "message" => "%{HTTPD_COMMONLOG}" }
     }
}
output {
     csv {
         path => "/home/output/httpd_file.csv"
         fields => ["clientip" ,"verb", "bytes"]
         csv_options => {"col_sep" => "    "}
     } 
     stdout{
        codec => rubydebug
    }
} 
```
####3.2 步骤  
1）进入logstash bin目录  
```
cd /bigData/tools/logstash/bin
```
2）启动logstash  
```
logstash -f /home/config/logstash-filebeat-csv.conf
```
####4、采集结果验证  
___
根据logstash里的输出路径配置  
```
/home/output/httpd_file.csv
```
####5、logstash 调试  
___
/home/config/grok-debug.conf  
```
input{
    stdin{}
}

filter {
  # [21:17:06:164] [INFO] - com.study.spring.ch01.controller.HelloWorldController.handleRequest(HelloWorldController.java:23) - into helloworld controller
  grok {
      match => [
          "message", "\[(?<timestamp>[\d:]+)\] \[(?<level>[\S]+)\] - (?<classMethod>[\S]+) - (?<info>[\S\s]*)"
      ]
  }
}


output{
   stdout{
        codec => rubydebug
   }
}
```  
